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Distributed  programming  is  characterized  by  high  communications  costs 


*MOD  is  a  high-level  language  system  which  attempts  to  address  these  prob' 


lems  by  creating  an  environment  conducive  to  efficient  and  reliable  network 


software  construction.  The  concept  of  a  processor  module  is  introduced  as 


well  as  a  methodology  for  distributed  data  abstraction  and  process  communica 


tion.  In  addition,  a  VHLN  (virtual,  high-level  language  network)  is  proposed 


AMS(MOS)  Subject  Classification;  68A05,  68A10,  68A55 


Key  Words:  distributed  programming,  modula,  processor  module, 


data  abstraction,  programming  languages 


Work  Unit  #8  (Computer  Science) 


Sponsored  by  the  United  States  Army  under  Contract  No.  DAAG29-75-C-0024  and  the 


Ccanputer  Sciences  Department,  University  of  Wisconsin,  Madison 


SIGNIFICANCE  AND  EXPLANATION 


A  language  (*MOD)  for  distributed  programming  is  being 
designed  and  implemented  at  the  University  of  Wisconsin  to 
facilitate  research  in  network,  software  concepts.  Distributed 
programming  is  characterized  by  the  use  of  multiple  hardware 
processors  to  implement  an  algorithm.  The  *MOD  system  also 
proposes  a  convenient  methodology  for  the  debugging  and  develop¬ 
ment  of  distributed  prograuns.  In  addition,  the  language  contains 
some  new  approaches  in  the  areas  of  data  abstraction,  mutual 
exclusion,  and  synchronization. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


*MOD — A  LANGUAGE  FOR  DISTRIBUTED  PROGRAMMING 


Robert  p.  Cook 


1 .  Introduction 

♦I'lOD  ( star  mod)  is  a  language,  derived  from  '1odula[27],  which 
is  intended  for  systems  programming  in  the  network  environment. 
The  *1^00  project  is  based  on  experience  with  our  PDPll/VAX  'lodula 
compilar[4]  and  was  inspired  by  Brinch  Hansen's  "distributed 
processes"  concepts[3].  The  design  strives  to  address  the  sys¬ 
tems  programmer's  traditional  concern  for  efficiency  and  includes 
the  constraint  that  each  language  feature  should  be  maximally  ex¬ 
tensible.  For  example,  the  ♦MOD  user  can  utilize  the  data 
abstraction  mechanisms  to  construct  either  queue  or  stack  types; 
thus,  the  language  attempts  to  define  an  appropriate  set  of  prim¬ 
itives  which  can  be  extended  to  meet  programming  needs.  By  giv¬ 
ing  each  user  the  freedom  to  experiment  with  language  constructs 
for  distributed  programming,  *M0D  is  also  intended  as  a  mechanism 
for  research.  This  paper  discusses  the  rationale  behind  the 
design  of  the  *M0D  system  and  contrasts  the  language  features 
chosen  with  those  of  the  Department  of  Defense(DoD) 
language! 151 ,  Hoare's  Communicating  Sequential  Processes 
{CSP)[14],  Feldman's  PLITS[2,9],  and  Brinch  Hansen's  "distributed 
processes" [ 3 1 .  In  particular,  we  address  the  distributed  pro¬ 
gramming  problem  areas  of  interprocessor  communication,  type 
checking,  separate  compilation,  debugging,  and  kernel  efficiency. 
The  *M0D  language  def inition[51  should  be  consulted  for  the  de¬ 
tails  of  design  decisions  in  other  areas  such  as  data  abstraction 
or  synchronization. 


Sponsored  by  the  United  States  Army  under  Contract  No,  DAAG29-75-C-0024  and  tho 
Computer  Sciences  Department  University  of  Wisconsin,  Madison. 


2 .  Systam  Ovsrview 

Before  proceeding  further  with  a  more  detailed  discussion  of 
the  distributed  programming  features,  we  will  consider  the  module 
concept  of  Modula  as  a  focal  point  for  program  development.  ^ 
module  encapsulates  an  environment  and  defines  the  relationship 

between  itself  and  the  outside  world;  therefore,  both  the 
information-hiding  properties  proposed  by  Parnas[231  and  the 
flexibility  of  the  Simula[6]  "class"  mechanism  are  maintained. 
Each  module  usually  corresponds  to  a  program  abstraction  and  con¬ 
sists  of  an  external  interface  specification,  data  structure  de¬ 
finitions,  procedures,  processes,  and  an  optional  initialization 
part. 

A  computer  system  is  traditionally[25]  viewed  as  a  collec¬ 
tion  of  processors,  processes,  and  procedures.  A  processor  exe¬ 
cutes  commands  or  instructions,  a  procedure  is  a  sequence  of  in¬ 
structions  for  a  processor,  and  a  process  is  one  or  more  pro¬ 
cedures  together  with  the  information  necessary  to  control  and  to 
define  the  virtual  processor  on  which  it  runs.  provides 
these  entities  in  the  forms  of  a  "processor  module",  procedure 
and  process  declarations,  respectively.  In  addition,  a  "network 
module"  is  required  to  define  system  connectivity  for  the  proces¬ 
sors  and  to  declare  any  global  types  or  constants.  These  module 
types  can  be  declared  with  the  following  syntax: 
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•>10DULEDECL^RMI0N:  = 


MOOULETYPE;  [external;  ] 

[define  ELEMENT  [, ELEMENT  1...;  ] 

[export  ELEMENT  [, ELEMENT  ]...;  1 
[pervasive  ELEMENT  [, ELEMENT  ]...;] 

[BLOCK  ] 

end  IDENTIFIER 

MODULETYPE := 

[ interface  1  processor  1  module  IDENTIFIER! 

network  module  IDENTIFIER=LINK  [,LINK  ]... 

BLOCK := 

[import  IDENTIFIER  [, IDENTIFIER  ] 

[DECLARiiTIONLIST  1 
begin  ST\TEMENTLIST 

LINK:=  (PROCESSORID  [ , PROCESSORID 

ELEMENT:=  IDENTIFIER  [ (PROCEDDREID)  1 

The  IDENTIFIER  names  the  module  and  must  be  matched  by  the 
IDENTIFIER  at  the  end  of  the  BLOCK.  The  BLOCK  consists  of  de¬ 
clarations  for  constants,  types,  variables,  modules,  processes, 
or  procedures  as  well  as  a  ST\TEMENTLIST  which  can  be  used  to  in¬ 
itialize  the  module.  The  module  boundary  delineates  a  closed 
lexical  scope  which  can  only  be  superseded  by  the  explicit 
specification  of  "define",  "export",  or  "import"  lists. 
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An  IDENTIFIER  specified  in  an  "import"  list  ceases  a  de¬ 
claration  from  a  global  scope  to  be  made  accessible  within  the 
module.  The  "export"  attribute  allows  a  local  declaration  to  be 
visible  at  the  enclosing  lexical  level;  while  "pervasive"  makes 
the  IDENTIFIER  known  at  all  lexical  levels  where  the  same  name  is 
not  already  declared.  The  latter  option  is  most  useful  for 
system-wide  constant  and  type  definitions.  The  optional  PRO- 
GEDUREID  can  be  used  to  specify  automatic  initialization  for  ex¬ 
ported  types.  The  "define"  statement  is  provided  as  an  alterna¬ 
tive  to  "export".  It  gives  the  user  the  ability  to  list  those 
IDENTIFIERS  which  can  be  referenced  externally,  but  only  by  pre¬ 
fixing  the  reference  with  the  module  name  as  with  the  Simula 
"class"  notation.  Furthermore,  the  "define",  "export",  and  "per¬ 
vasive"  statements  provide  implicit  read-only  protection  for  any 
variable  so  listed.  The  ability  to  specify  the  external  inter¬ 
face  for  each  module  is  becoming  a  standard  feature  of  modern 
programming  as  is  demonstrated  by  its  use  in  Mesa[101,  Eu- 
clid[13],  Alphard[24],  ADA[15],  etc. 

Each  processor  LINK  specifies  a  list  of  processor  modules 
which  can  be  sent  messages.  No  variables  or  shared  code  are  al¬ 
lowed  at  the  network  level;  any  procedures  used  for  type  imple¬ 
mentations  are  replicated  in  the  appropriate  "processor  module"s. 
Each  "processor  module"  can  represent  any  number  of  physical  pro¬ 
cessors  as  long  as  they  all  use  a  shared  memory  for  instruction 
execution.  The  external  interface  specification  for  a  "processor 
module"  lists  any  message  types  and  process  names  which  are  used 
for  communication.  We  should  also  point  out  that  the  availabili¬ 
ty  of  a  hardware  multiprocessor  to  implement  a  particular  "pro¬ 
cessor  module"  should  be  regarded  as  a  fortuitous  circumstance 
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and  should  not  ba  counted  on  by  the  programmer.  ^n  "interface 
module"  is  a  ''lodula  constructr271  ,  similar  to  a  monitor[l3], 
which  guarantees  mutual  exclusion  across  all  the  contained  pro¬ 
cedures. 

The  example  in  Figure  1  illustrates  these  concepts  with  a 
ring  network  version  of  Dijkstra's  Dining  Philosophers[7 ]  prob¬ 
lem.  As  in  the  original  version,  five  philosophers  are  each  try¬ 
ing  to  eat  a  plate  of  special  spaghetti  which  has  been  placed  in 
the  middle  of  a  round  table.  In  our  example,  each  philosopher 
can  only  directly  control  the  right-hand  fork;  to  gat  the  left 
fork,  the  philosopher  to  the  left  must  be  consulted.  However, 
each  philosopher  is  also  restricted  to  conversation  with  the 

right  neighbor  only;  therefore,  a  message  must  be  sent  around  the 
ring (table)  to  get  permission  to  use  the  left-hand  fork  and  to 
give  it  back.  The  algorithm  is  based  on  an  ordered  resource  al¬ 
location  strategy  developed  by  Havender[I2]  which  prevents 
deadlock  and  starvation. 

The  "diningroom"  network  definition  specifies  the  connec¬ 
tivity  for  the  ring  network,  defines  a  "semaphore"  data  type,  and 
contains  the  program  stubs  for  the  five  philosophers.  Since 
"samaphore"s  are  declared  as  "pervasive",  they  will  appear  as 
builtin  types  in  all  processors  in  which  "semaphore"  is  not  rede- 
clared.  The  five  processors  are  specified  only  in  terms  of  their 
external  interfaces  (termed  a  program  stub  ).  The  keyword 
"external"  indicates  that  the  processor  definitions  are  part  of  a 
separate  compilation.  The  binding  process  for  separate  compila¬ 
tions  will  be  discussed  in  Section  1. 


Figure  1 


network  module  dining room= (phi IT , phill ) , (phill , phi 12) , (phil2 , phi  13 )  , 

( phi 13 ,phil4 ) , (phil4 , phi Id ) ;  (*ring  network*) 
interface  module  semaphoredef ;  (*Boolean  semaphore  abstraction*) 

pervasive  semaphore ( init) ,P,V;  (*available  to  all  processors*) 
type  semaphore  =  record  taken:  boolean; 

free  :  s iqnal ; 
and  record; 

procedure  P (var  s:  semaphore); 
beg  in 

i f  s. taken  then  wait(s.free)  end  if; 
s . taken : =true 
end  P; 

procedure  V(var  s:  semaphore); 
begin 

s.takan;=false;  send(s.free) 
end  V; 

procedure  init (var  s:  semaphore); 
beg  in  s.takan:  =  false 
end  init; 
end  semaphoredef; 

processor  module  phi 13;  external; 

define  get, put, got;  (*program  stub  for  philosopher  zero 

process  get (who , fork :  integer) ;  (*get  "fork"  for  "who"  *) 
process  put(fork:  integer) ;  (*give  "fork"  back  *) 

process  got(who:  integer) ;  (*tell  "who"  the  news  *) 

(*refarencad  externally  as  philfl.get,  etc.  *) 
end  phild; 

processor  module  ph i 1 4 ;  external ; 


end  ph i 1 4 ; 
end  diningroom. 


3 .  Language  Concepts 

From  the  ♦MOD  viewpoint,  a  computer  network  can  be  charac¬ 
terized  as  an  arbitrary  collection  of  processors  with  fixed  com¬ 
munication  paths  for  interprocessor  message  transfer.  Messages 
are  assumed  to  range  from  no  content(signal  or  interrupt)  to  ar¬ 
bitrary  data  structures.  Furthermore,  we  require  strong  type 
checking  both  within  and  across  processors  to  maintain  system 
consistency.  Finally,  any  mechanisms  presented  should  be  effi¬ 
cient  and  should  not  constrain  the  options  of  the  systems  pro¬ 
grammer.  For  these  reasons,  we  developed  a  process-oriented  com¬ 
munication  methodology  which  eliminated  the  need  for  additional 
statements  to  handle  messages.  In  the  next  sections,  the  *MOD 
design  will  be  presented  along  with  a  detailed  discussion  of  the 
alternatives,  advantages  and  disadvantages. 

3 • 1  Processe s  and  Signals 

Each  "processor  module"  consists  of  one  or  more  concurrent 
processes  declared  as  follows: 

PROCESSDECL^RATION; = 

process  IDENTIFIER  [ (FORMALS)  1 

[  '['expression']'  ]  [:TYPEID]; 

BLOCK 

end  IDENTIFIER 

PROCESSREFERENCE:= 


PROCESSID  [ (ARGUMENTLIST)  ] 


Exc3pt  for  the  keyword  "process"  and  the  optional  priority  EX¬ 
PRESSION,  the  declaration  is  identical  to  that  of  a  procedare; 
however,  the  semantics  are  different  since  a  process  can  execute 
independently  of  its  creator.  Each  instance  of  a  process  is 
created  by  a  PROCESSREFERENCE  which  must  specify  a  list  of  argu¬ 
ments  corresponding  exactly  in  type  and  number  to  the  F0RMAL3. 
At  this  point,  storage  space  is  obtained  for  the  activation 
record  and  the  process  control  block,  both  of  which  remain  allo¬ 
cated  until  the  process  terminates.  The  returned  value  for  a 
functional  process  is  set  by  assignment  to  the  process  identifier 
and  must  match  the  specified  TYPEID.  When  a  functional  process 
exits,  the  returned  value  is  copied  from  its  activation  record  to 
the  address  space  of  the  caller.  The  use  of  a  functional  process 
corresponds  to  sending  a  message  and  then  waiting  for  a  reply 
while  a  reference  to  a  non-functional  process  implies  parallel 

execution.  We  will  frequently  use  the  term  message  as  a  synonym 
for  the  record  containing  the  arguments  to  or  result  from  a  pro- 

C 0S S  • 

The  optional  EXPRESSION  must  evaluate  to  a  compile-time  con¬ 
stant  which  specifies  the  initial (defaul t  zero)  priority  of  the 
process.  Each  process'  priority  can  be  modified  by  assignment  to 
the  variable  "priority"  which  is  used  to  control  context  switches 
among  processes.  The  general  rule  is  that  a  process  loses  con¬ 
trol  of  the  hardware  processor  if  it  lowers  its  priority  below 
that  of  another  "ready"  process  or  if  a  higher  priority  process 
changes  to  the  "ready"  state.  The  other  builtin  process 
ident i f i er ( "or ig in" )  is  a  time  stamp  which  indicates  the  creation 
order  relative  to  all  other  processes  in  the  same  processor 


module.  The  example  in  Figure  2  completes  the  Dining  Philoso¬ 
phers  network  by  defining  the  actions  of  each  processor  and 
serves  as  an  illustration  of  the  preceding  definitions. 

Each  philosopher  is  required  to  request  the  forks  in  a 
specific  order  and  must  have  obtained  the  first  fork  before  re¬ 
questing  the  second.  The  "get"  process  accepts  fork  requests  and 
either  passes  the  request  to  the  right  in  the  ring  or  else  gats 
control  of  the  fork  and  sands  an  acknowledgment  to  the  "who"  phi¬ 
losopher.  The  "got"  process  uses  its  higher  priority  to  speed 
the  acknowledgments  to  the  appropriate  philosophers.  Finally,  it 
should  be  noted  that  multiple  activations  of  each  process  can 
coexist;  for  instance,  three  independent  copies  of  the  "get"  pro¬ 
cess  could  be  handling  requests  simultaneously. 

The  signal  construct  embodies  a  message  capability  that  is 
even  simpler  than  a  process  call  in  that  its  arrival  represents 
the  only  content.  "signal"  can  be  used  as  a  basic  type  in  *>100 
to  declare  variables  which  can  only  be  manipulated  by  the  follow¬ 
ing  procedures: 

send (SIGN^LID)  wa i t (SIGN^LTD, RANK) 
awaited (SIGNALID) 

The  interpretation  of  these  procedures  is  identical  both  within 
and  across  processors.  A  "wait"  delays  the  executing  process  in 
a  priority  queue  specific  to  5IGNALID.  The  queue  is  or¬ 
dered  first  by  the  RANK  attribute  and  secondly  by  the  longest 
wait  time.  The  "awaited"  function  returns  a  Boolean  value  which 
reflects  the  status  of  the  "wait"  queue ( true=not  empty)  for  313- 
NALID.  The  "send"  statement  unlinks  the  process  at  the  head  of 
SIGNALID's  queue  and  sets  the  process'  status  to  "ready".  If  no 
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>rocss3o r  .Tiodula  phil..  (*11^1^4*) 

ins  gst,put,got;  (*n=ifl  mod  5,  right  neighbor*) 

import  phil  ; 

const  Ei  rst=nun(  i  ,  i  1-4  mod  5); 

second=max  ( i ,  i-H4  mod  5); 
var  myfcrk ,gotit :  semaphors; 

process  get (who , fork :  integer) ;  (*gst  fork  for  who*) 

beg i n  (*calls  who  on  success*) 

U  fork  <>  i  then  phil  get (who, fork) ; 

else  P(myfork);  got(who); 
end  if; 

end  g e t ; 

process  put(fork:  integer) ;  (*givs  "fork"  back*) 

begin  (*wake  up  anyone  waiting*) 

i_f  fork  <>  i  then  phil  put  ( fork)  ; 

else  V(mytork) ; 
end  i f ; 

and  put; 

process  got(who:  i ntager)  ri);  (*lat  "who"  use  "fork*) 
beg  in 

it  who  <>  i  then  phil  got (who)  ; 
else  V (got i t) ; 
end  i f ; 

end  got; 

procedure  g e t fo r k ( f o r k ;  integer) ;  (*philosophor  waits  for 
begin  get  (  i  .fork)  ;  P(gotit); 
and  getfork; 
beg i n  loop 

(*think*)  getfork(f irst) ;  getfork ( second) ; 

(*eat*)  put(first);  put(second); 
and  loop; 
end  phil.. 


process  is  waiting  for  the  signal,  the  "send 


is  ignored. 


The 


semaphore  example  in  Figure  1  uses  both  the  message  and  delay 
capabilities  of  signals  to  build  a  synchronization  primitive. 

For  interprocessor  communication,  the  "import"  processor  can 
only  perform  "send"  operations  on  a  signal  while  the 
"export/define"  processor  is  unrestricted.  Thus,  a  signal  can  be 
thought  of  as  a  means  of  generating  a  name  for  a  processor  which 
also  embodies  a  communication  capability.  The  capability  is  also 
revocable  in  the  sense  that  if  the  defining  processor  never 
"waif's  for  the  signal,  the  signal  will  always  be  ignored  by  de- 
f ini tion. 

3 . 2  Design  Decisions 

One  of  the  major  differences  of  opinion  in  distributed  pro¬ 
gramming  language  design  occurs  over  the  use  of  process-oriented 
versus  message-oriented  communication.  The  DoD  ^DA  language  is  a 

typical  example  of  the  latter  choice.  In  a  recent  paper  by  Lauer 
and  Needh3m[l9],  they  state  "that  these  two  categories  are  duals 
of  each  other  and  that  a  system  which  is  constructed  according  to 
one  model  has  a  direct  counterpart  in  the  other."  We  agree  with 
the  duality  conclusion  but  feel  that  the  message-oriented  ap¬ 
proach  has  soma  deficiencies  for  our  purposes;  these  deficiencies 
may  be  completely  irrelevant  in  other  applications.  The  follow¬ 
ing  example  from  ADA[16]  will  illustrate  most  of  these  points. 
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task  SE''1APaORE  is 


task  body  SE.'IAPHORE  is 


antry  P;  bag i n  loop 

antry  V;  accept  P; 

and;  accept  V ; 

end  loop; 

end; 

initiate  SEMAPHORE; 

P; 

<critical  section> 

V; 

The  "initiate"  statement  causes  the  creation  of  the  3EM^- 
PHORE  task  which  is  equivalent  to  a  *.MOD  process;  note  that  a 
second  "initiate"  while  SEMAPHORE  is  still  active  would  be  an  er¬ 
ror.  The  "accept"  statement  defines  a  message  reception  point 
within  a  task  as  follows: 

accept  ENTRYNAME  [(F0RMAL3)  1 
[do  STATEMENTS 
and  [ENTRYNAME  1  1 ; 

The  procedure  call  syntax  is  used  to  send  a  message  to  an  "ac¬ 
cept"  point.  The  "accept"  statement  must  be  executed  by  an  ac¬ 
tive  task  to  receive  a  message.  The  execution  semantics  are  as 
follows.  "Whichever ( receiver  or  sender  task)  gets  there  first 
waits  for  the  other.  When  the  rendezvous  is  achieved,  tlie  ap¬ 
propriate  parameters  of  the  caller  are  passed  to  the  called  task 
...  The  caller  is  then  temporarily  suspended  until  the  called 


task  completes  the  statements  embraced  by  do  ...  end .  \ny  oat 
parameters  are  then  passed  back  to  the  caller  and  finally  both 
tasks  proceed  independently  of  each  other." [16] 

Since  "accept"  is  an  executable  statement,  the  receiving 
task  can  choose  the  execution  point  at  which  to  receive  the  mas¬ 
sage;  thus,  mutual  exclusion  among  competing  messages  is  provided 
automatically.  Also,  the  receiving  task ( process)  remains  static 
while  a  *MOD  process  is  activated  for  each  message.  Other  useful 
properties  of  the  message-oriented  approach  are  listed  in  Lauer 
and  Needham[19].  However  in  our  opinion,  the  message-oriented 
approach  has  the  following  disadvantages  for  a  systems  program¬ 
ming  language. 

Message  Queuing.  Since  the  receiving  task  in  a  message  system 
can  only  process  one  message  at  a  time,  the  kernel  must  queue  any 
additional  messages  which  arrive  before  the  next  "accept"  state¬ 
ment  is  executed.  In  *MOD,  message  queuing,  stacking,  etc.  are 
choices  under  user  control  since  every  message  has  a  process  to 
implement  the  delay  protocol.  Several  examples  of  the 
*MOD/Modula  programming  approach  may  be  found  in  Wi  r th  [ 7.3 , 33 ]  or 
in  an  application  description  by  Andrews[l]. 

\ctive  Processes.  In  *MOD,  the  recipient  of  a  message  is  normal¬ 
ly  pa3si'/e(no  activation  record  or  process  control  block)  ;  in  the 
massage-oriented  approach,  the  task  is  always  active  although  it 
may  be  delayed  at  an  "accept"  statement.  Consider  the  semaphore 
example.  Every  semaphore  used  in  an  operating  system  would  be  an 
independent  task  with  an  activation  record  and  control  block. 
This  fault  is  corrected  in  *MOD  by  separating  message  transmis¬ 
sion  and  synchronization  facilities.  In  addition,  the  * dOO  user 
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can  always  create  a  static  process  but  the  user  of  a  message- 
oriented  system  can  never  create  a  passive  process  which  can  "ac¬ 
cept"  a  message. 

Delayed  Processes.  The  caller  is  always  suspended  until  the 
called  task  completes  the  entry  routine.  This  defect  is  present 
in  both  ADA  and  "distributed  processes"  but  not  in  PLITS,  *MOD, 
or  Lauer  and  Needham.  The  primary  reason  for  delay  is  to  wait 
for  a  returned  message.  *'^0D  provides  all  combinations  of  "delay 
until  called  process  terminates"  and  "delay  until  called  process 
returns  a  message"  except  for  "no  delay  but  message  returned" 
which  can  be  easily  programmed. 

Process  Deletion.  It  is  dangerous  to  delete  processes  because  of 
possible  late  arrivals  or  of  the  potential  loss  of  messages  in 
the  wait  queue.  This  is  not  a  problem  in  *MOD  since  every  mes¬ 
sage  is  guaranteed  a  corresponding  process. 

Priority.  The  message-oriented  languages  have  no  way  to  force  an 
executing  task  to  recognize  a  high-priority  message.  If  the 
priority  were  associated  with  the  task  as  in  ♦MOD,  the  message- 
oriented  approach  could  distinguish  among  messages  to  competing 
tasks  but  not  among  different  "accept"  points  within  a  single 
task.  The  reason  is  that  even  if  the  task  could  recognize  the 
new  message,  it  would  be  illegal  to  duplicate  the  task  to  accept 
it.  Trying  to  associate  a  priority  with  the  message  is  also  fu¬ 
tile  because  there  is  no  way  to  force  the  executing  task  to  per¬ 
form  the  corresponding  "accept"  statement. 

"?•  Consider  the  "get"  process  in  Figure  2. 
If  three  simultaneous  messages  arrive,  three  independent  "get" 
processes  would  be  created  which  could  all  be  in  execution  on  a 
mul tiprocessor  system.  In  the  message-oriented  approach,  each 
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tas!^  can  process  ona  message  at  a  time  and  it  is  illegal  to  du¬ 
plicate  a  task;  therefore,  software  is  automatically  biased  to¬ 
ward  single  processor  systems.  In  *MOD,  the  distinction  was  in¬ 
tended  to  be  transparent. 

Synchronization  Primitives.  The  ^DA  semaphore  example  is  typical 
of  a  synchronization  primitive  constructed  using  messages.  The 
disadvantage  is  that  a  task  must  be  created  for  each  semaphore. 
Additional  problems  arise  if  the  order  of  message  arrival  forms 
the  basis  for  synchronization.  Since  separata  queues  are  main¬ 
tained  for  parallel  "accept"  statements,  the  sequence  information 
is  lost.  In  all  fairness,  we  will  also  agree  that  synchroniza¬ 
tion  primitives  are  less  useful  in  a  message  system  since  the 
"accept"  statement  is  an  exclusion  mechanism. 

3 .  3  Other  Languages 

Hoare's  CSP,  DoD's  ADA,  and  Feldman's  PLITS  language  are  all 
message-oriented.  CSP  and  PLITS  are  oriented  toward  end-user 
programming.  PLITS,  for  instance,  performs  automatic  routing  of 
messages  which  would  be  a  user- implemented  service  in  *MOD.  CSP 
is  strongest  in  its  exploration  of  nondeterministic  programming 
features  while  PLITS  is  more  completely  specified  with  respect  to 
distributed  programming.  The  ADA  language  is  a  CSP  derivative 
which,  if  used  for  systems  programming,  suffers  from  the  defects 
listed  earlier. 

extends  Brinch  Hansen's  "distributed  processes"  for 
network  communication  and  improves  the  multiprogramming  features 
of  Wirth's  dodula  language.  For  example,  each  "distributed  pro¬ 
cess"  can  be  encoded  as  a  * '10D  "processor  module"  as  follows: 
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process  distributed 
<own  variables> 

proe  narae(  input  paramsl^output  params) 

<local  variables> 

<statement> 

<initial  statement> 

processor  module  *MODd istr ibuted; 
define  name,  nametype; 

<own  variables> 
type  nametype=  record 

output  params 
end  record; 

process  name( input  params) ;nametype; 

<local  variables> 
begin  <statemant> 
end  name; 

beg  in  <initial  statement> 
end  ♦I'lODd  istr  ibuted  . 

The  *MOD  extensions  to  these  languages  are  summarized  below 
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1.  Added  processor  and  network  module  notation. 

2.  Defined  network-wide,  strong  type  checking. 

3.  Defined  a  separate  compilation  facility  for  network  construc¬ 
tion. 

4.  Added  "size"  specifications  to  types  for  interface  defini¬ 
tions. 

5.  Deleted  "device  modules". 

6.  Integrated  priority  into  signal  and  process  handling. 

7.  Changed  signal  and  process  semantics  to  conform  to  network 
usage. 

3.  Improved  external  interface  specification  statements. 

9.  Added  data  abstraction  primitives  and  parameterized  types. 

In  addition,  we  have  retained  a  "degree  of  transparency" [2 1] 
between  concurrent  and  distributed  programming  features  while 
trying  to  satisfy  the  efficiency  constraints  of  a  systems  pro¬ 
gramming  language. 

^ •  System  Construction  and  Testing 

*MOD  has  more  severe  binding  problems  than  found  in  most 
systems  in  that  a  network  can  be  separately  compiled  by  processor 
modules,  processor  modules  by  modules,  and  modules  by  even  small¬ 
er  modules.  All  must  be  type  checked  and  loaded  to  form  a  work¬ 
ing  system.  The  compilation  order  rule  is  that  all  declarations 
must  be  available  prior  to  a  module's  compilation.  A  module  with 
the  "external"  clause  is  termed  a  program  stub  which  can  be  de¬ 
fined  by  the  programmer  or  created  by  the  *MOD  compiler.  The 
former  option  can  be  used  for  top-down  program  creation  or  to  in¬ 
terface  with  foreign  systems.  Once  the  corresponding  module  body 
is  defined  as  in  Figure  2,  the  ♦MOD  compiler  automatically  gen- 
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arates  a  program  stub  in  textual  form.  There  are  saveral  advan¬ 
tages  to  this  approach. 

First,  the  programmer  only  maintains  one  object--the  pro¬ 
gram;  this  may  be  contrasted  with  other  languages  which  require 
the  user  to  maintain  both  a  declarations  package  and  a  program 
body.  Since  the  *MOD  stub  is  generated  from  the  most  recent  copy 
of  the  program,  the  probability  of  inconsistencies  between  the 
two  definitions  is  minimal.  Finally,  the  *MOD  method  automati¬ 
cally  generates  a  program  specification  in  a  machi ne- independent 
format  which  is  useful  if  software  is  being  developed  at  several 
different  sites. 

Once  the  object  modules  for  a  processor  module  have  been 
created,  a  program  called  the  binder  is  invoked  which  checks  the 
creation  dates  for  consistency  and  releases  the  object  modules  to 
the  system  loader  to  create  a  bootable  core  image  file.  The  core 
image  contains  the  machine-dependent  *MOD  kernel  discussed  in  the 
next  section  and  a  compiler-created  table  which  contains  the  mes¬ 
sage  codes  for  external  communication  and  the  process  addresses 
for  arriving  transmissions.  ^t  this  point,  the  software  can  be 
tasted . 

In  order  to  test  software  in  the  distributed  environment,  it 
must  be  possible  to  experiment  both  with  software  algorithms  and 
hardware  components.  The  advent  of  high-level  languages  has 
greatly  enhanced  algorithm  development  but  the  same  flexibility 
is  not  present  for  hardware.  The  virtual  machine  approach  [111 
was  a  step  in  the  right  direction  but  was  primarily  oriented  to¬ 
wards  the  construction  of  mul ti prog  rammed ,  single  processor 
software.  The  VM  environment  has  been  suggested  [26]  as  suitable 
for  the  development  of  network  software  but  the  user  is  still 
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giv3n  a  bars  machine  as  a  starting  point. 

Our  proposal  consists  of  a  two-lavel  approach--  1)  a 
VHLN (virtual  high-level  language  network)  *MOD  environment  for 
network  software  development  and  experimentation  on  a  single  host 
computer  (a  PDPll  or  V\X  in  the  current  implementation  effort); 
and  2)  a  compiler  capable  of  producing  bootable  code  for  a 
number  of  different  machines.  The  VHLN  system  is  a  simulated 


n  e  t  wo  r  k 

which  uses  compiled  code 

and 

runtime  aids 

to 

provide  a 

variety 

of  error-checking 

and 

debugging 

aids 

.  Thus, 

hardware/software  systems  can 

be 

developed  in 

an 

economical 

manner  on  a  single  processor,  even  if  the  target  hardware  is  not 
available  due  to  delivery  or  design  problems.  The  *MOD  system 
can  also  be  used  as  a  research  tool  by  institutions  that  only 
have  a  single  processor.  Once  a  software  system  has  been  tested, 
it  can  be  moved  to  the  host  machines  for  production  use. 

Several  advantages  ensue  from  the  use  of  a  VHLN  environment. 
First,  as  is  illustrated  in  the  Dining  Philosophers  example, 
there  are  no  machine  details  to  muddle  up  the  solution.  In  fact, 
if  machine  designers  made  peripheral  devices  maintain  the 
process/signal  philosophy,  no  such  bit  twiddling  would  ever  be 
necessary!  The  high-level  language  methodology  provides  system- 
wide  type-checking,  data  abstraction,  and  encapsulation  mechan¬ 
isms.  In  addition,  the  "defining  module"  concept  insures  that 
someone  is  responsible  for  every  bit  of  software  in  the  system; 
if  something  does  not  work,  the  culprit  is  easily  found.  41so, 
software  can  be  developed  in  an  environment  that  provides  a  use¬ 
ful  set  of  diagnostic  and  debugging  tools  before  being  moved  to 
the  target  computer  system.  It  is  much  more  economical  to 
deeelop  software  aids  for  one  host  development  computer  than  for 
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aach  target  machine.  The  VHLN  system  also  has  some  disadvan¬ 
tages  . 

The  simulated,  multiple  processor  environment  provided  by 
the  *MOD  system  is  unrealistic  in  the  sensa  that  events  will  not 
have  a  real-time  correspondence  to  the  performance  of  systems 
running  on  bare  machines.  It  might  be  possible  to  add  some 
clock-driven  primitives  to  the  language  to  address  this  problem 
if  it  proves  serious.  As  we  gain  more  experience  with  the  sys¬ 
tem,  it  will  be  possible  to  draw  more  definitive  conclusions  re¬ 
garding  the  ease  of  moving  from  a  simulated  to  a  real  network. 

5.  The  *MOD  Kernel 

The  *MOD  kernel  performs  process  control  and  message 
transmission  functions  only;  any  routing,  security,  buffering,  or 
flow  control  operations  are  the  domain  of  the  systems  programmer. 
In  addition,  process  control  does  not  include  the  traditional 
scheduling  decisions.  A  process  loses  control  of  a  hardware  pro¬ 
cessor  only  by  terminating,  blocking,  lowering  its  priority,  or 
by  receipt  of  a  message  for  a  higher  priority  process.  Therefore, 
the  major  kernel  function  is  message  processing  which  is  con¬ 
trolled  by  a  compiler-generated  table. 

A  list  is  generated  by  the  compiler  which  details  the 
processes  referenced  and  the  corresponding  processor  identifica¬ 
tion;  in  addition,  a  similar  list  is  created  for  local  processes 
that  occur  in  "export"  or  "define"  statements.  When  an  external 
process  request  occurs,  the  kernel  constructs  a  massage  from  the 
argument  list  as  follows: 

var  message:  record 

arg^:  t^; 
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3r9N:  T(j; 
end  record 

The  processor  identification  for  the  destination  is 
from  the  kernel  tables  so  the  message  is  transmitted  along  with 
some  control,  error,  and  sequence  information.  When  the  message 
is  received,  the  target  processor  verifies  that  the  process  has 
been  exported,  creates  an  activation  record  containing  the  mes¬ 
sage  for  the  new  process,  and  sets  the  "origin"  time  stamp.  If 
the  new  process  has  the  highest  "ready"  priority,  it  will  begin 
execution  immediately.  For  a  functional  process  call,  the  ori¬ 
ginating  kernel  changes  the  process  status  of  the  caller  to  indi¬ 
cate  that  it  is  waiting  for  a  reply  message  from  the  target  pro¬ 
cessor.  When  a  reply  arrives,  its  origin  is  verified  against 
that  in  the  process'  control  block;  the  reply  is  appended  to  the 
process'  activation  record;  and  the  process'  status  is  changed  to 
"ready".  It  will  resume  execution  if  it  has  the  highest  priori¬ 
ty. 

^t  this  point,  it  should  be  noted  that  most  I/O  processors 
do  not  have  the  sophistication  or  the  flexibility  to  implement 
the  previous  protocol.  Wirth  [27]  proposed  the  concept  of  a 
"device"  module  to  capture  the  essence  of  I/O  programming.  3y 
necessity,  the  device  module's  syntax  is  vary  machine  and,  in 
some  cases,  device  dependent.  For  instance,  a  PDP-11  module  con¬ 
tains  priority  level,  interrupt  location,  and  device  register 
syntax.  Device  modules  have  been  omitted  from  *^0D  because  of 
their  inherent  machine  dependency;  instead,  process  calls  and 
signals  are  used  to  their  fullest  capabilities.  The  advantage  is 
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that  all  programs  retain  their  machine  independence  at  the  ex¬ 
pense  of  a  small  number  of  machine  language  instructions  for  each 
processor  >rfhich  convert  device  interrupts  to  signals  or  process 
calls,  as  appropriate.  Consider  the  following  example  from  wirth 
[23]. 

device  module  timing [6]; 
define  tick; 
use  time; 

'/ar  tick:  signal;  LCS  [177546bl  ;  bits; 
process  d r  i  ver [ 1 00 b] ; 
beg  i  n 

LCS [5 ] : =tr ue ; { *star t  clock  running*) 
loop 

do  io ;  i  nc  ( time)  ; 
while  awa i ted ( t ick)  do 
send ( t ick) ;  and; 
end  ( *loop* ) ; 
end  driver; 
beg  in  driver 
end  timing; 

rhi.3  example  would  be  expressed  in  *MOD  as  follows. 

process  timing; 

i  m  po  r  t  t  i  c  k  ,  t  i  m  e ; 
const  ST^RTCL0CK=4 ; 
var  interruptrsignal; 


pr ior i ty : =6 ; 

(*start  clock*) 
sys (STARTCLOCK, interrupt) ; 
loop 

wa i t ( inter  rapt) ;  i nc ( time) ; 
while  awa i ted ( tick)  do  send ( tick) ; 

end  while; 
and  loop; 
end  timing; 

The  "sys"  procedure  is  the  only  escape  mechanism  to  the  host 
machine.  In  the  example,  the  argument  ST^RTCLOCK  selects  a  ker¬ 
nel  routine  which  initializes  the  clock  interrupt  locat ion ( 1 00g ) 
and  starts  the  clock  running.  Since  the  *MOD  process  is  waiting 
at  priority  six,  the  "interrupt"  signal  will  wake  it  up  and  the 
priority  will  prevent  processor  preemption  by  messages  to  lower 
priority  processes.  The  priority  construct  provides  the  same 
Eunctional ity  as  processor  priority  and  has  the  advantage  of  be¬ 
ing  dynamic  as  opposed  to  the  static  notation  of  the  device 
module.  Another  advantage  of  the  high-level  approach  is  that 
device  handlers  can  be  easily  debugged  by  testing  their  response 
to  artificially  generated  signals.  Device  modules  have  one  ad¬ 
vantage  in  that  all  the  information  is  available,  but  force  the 
programmer  to  accede  to  the  perversity  of  the  hardware  designer 
(see  Wirth  [29]  for  a  list  of  typical  problems) .  Since  recent 
nardware  trends  indicate  a  growing  support  of  high-level  Language 
architecture,  there  is  no  reason  why  this  philosophy  should  not 
also  be  extended  to  external  communication  interfaces. 

This  implementation  also  makes  it  easy  to  program  hierar- 
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chies  of  interrupt  handlers  as  proposed  by  Lampson  [17].  Consid¬ 
er  the  folllowing  example  taken  from  Unix  [22]. 

The  Unix  system  is  initialized  so  that  a  memory  fault  trap 
goes  to  an  error  routine.  However,  to  create  the  table  of  free 
space  the  system  needs  to  know  the  current  memory  size.  This  is 
accomplished  by  doing  successive  stores  until  a  memory  fault  oc¬ 
curs;  the  point  of  the  fault  identifies  the  highest  available 
memory  address.  The  problem  occurs  in  redirecting  the  trap  away 
from  the  error  routine.  The  *MOD  solution  is  to  execute  a  "wait 
(memoryfaul t,2) "  which  supersedes  the  "wait  (memoryfaul t,  1) "  in 
the  error  routine.  When  the  signal  occurs,  the  trap  is  diverted, 
as  desired.  The  UNIX  solution  is  not  nearly  so  elegant. 

We  have  tried  to  illustrate  by  these  examples  that  the  dis¬ 
tinctions  between  I/O  programming  and  process  communication  are 
largely  artificial;  therefore,  one  methodology  will  suffice  for 
both . 


6.  Conclusions 

The  *MOD  system  represents  an  exploration  of  the  design  de¬ 
cisions  necessary  to  apply  the  modular  programming  philosophy  of 
Wirth  [30]  to  the  development  of  distributed  software  and  to  pro¬ 
pose  an  environment  conducive  to  the  construction  and  debugging 
of  such  systems.  This  paper  would  not  have  been  written  but  for 
the  impetus  and  inspiration  of  Brinch  Hansen's  excellent  arti- 
cle[3]  on  distributed  processes. 

The  current  *MOD  compiler  is  a  2300  line  C[23]  program  which 
runs  on  a  PDPll/45  or  V^X  UNIX  system,  is  one-pass,  generates  ob¬ 
ject  code,  and  compiles  at  3000  LPM(VAX)  or  1000  LP'l  ( 1 1/4  5 )  .  The 
compiler  is  table-driven  for  parsing,  semantic  analysis,  and  coda 
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generation  and  coaid  easily  be  modified  to  generate  code  for  oth¬ 


er  machines.  The  present  implementation  is  restricted  to  a  sin¬ 
gle  "processor  module"  and  has  been  well  tested  by  sixty  students 
each  of  whom  implemented  a  multiprogramming  operating  system 
which  executed  in  V>1  mode.  The  network  version  of  *MOD  is  under 


development . 
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